Image processing apparatus and method of controlling the same

ABSTRACT

This invention enables, for example, reduction of motion blur in a hold-type display device and reduce flicker in an impulse-type display device by a simple process. For this purpose, an LPF filters a frame of input image data (A[i]) to generate low-frequency image data (L). A subtractor and an adder generate high-frequency image data (SH). Another adder adds the low-frequency image data (L) from a delay circuit to subsequent low-frequency image data. A divider halves the sum to generate low-frequency averaged image data (SL). A switch alternately outputs the high-frequency image data (SH) and the low-frequency image data (SL) every time a frame of image data is input. As a result, the apparatus of this invention can generate output image data having a frame rate twice that of the input image data.

REFERENCE TO RELATED PATENT APPLICATION

This application is a continuation of U.S. application Ser. No.13/797,316, filed Mar. 12, 2013, which is a continuation of U.S.application Ser. No. 12/186,582, filed on Aug. 6, 2008, now U.S. Pat.No. 8,421,917, issued Apr. 16, 2013 which claims priority to JapanPatent Application No. 2007-207181, filed Aug. 8, 2007. The entiredisclosures of those applications are incorporated by reference herein.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing apparatus forconverting moving image data having a first frame rate into moving imagedata having a higher frame rate, and a method of controlling the same.

2. Description of the Related Art

Conventionally, a CRT has been synonymous with a moving image displaydevice for, for example, television. However, so-called liquid crystaldisplays, plasma displays, and FED displays have been put into practicaluse in recent years. That is, there are now displays of various types.

The displays of these types adopt different display methods. Forexample, display devices based on a liquid crystal device (e.g.,direct-view-type liquid crystal display device, liquid crystal rearprojector, and liquid crystal front projector) use many scanningmethods. In any case, the light output period in each pixel portionoccupies a large part of the display period of one frame. For thisreason, such a display device is called a hold-type display device.

On the other hand, in, for example, a CRT or FED, light is output ineach pixel portion once in a frame. The light emission time is muchshorter than the frame display period and is normally 2 msec or less.For this reason, such a display device is called an impulse-type displaydevice.

There also exist a so-called plasma display and a field sequentialdisplay which are of types different from the above-described classes.

The display methods of the respective types have the following features.

(1) Hold-Type Display Device

A display device of this type emits light during a large part of a frameperiod. Hence, the temporal imbalance of light intensity is small, andflicker is rarely observed. Additionally, pursuit (i.e., the pursuit bythe eyes of a moving portion of a moving image) makes motion blurrelatively large in accordance with the length of the light emissionperiod in a frame. “Motion blur” here is different from that caused bythe response characteristic of a display device.

(2) Impulse-Type Display Device

A display device of this type emits light in a very short time during aframe period. Hence, the temporal imbalance of light intensity is large,and flicker synchronous with a frame is observed. However, motion blurin pursuit is rarely observed. It is therefore possible to obtain aresolution almost equal to that of a still portion.

In general, the light emission period of a display device changesdepending on the display method and display device. The above-describedtypes (1) and (2) are diametrically opposed in terms of the lightemission period. The longer the light emission period (corresponding tothe hold time) in each method is, the larger the motion blur in pursuitis. The shorter the light emission period is, the smaller the motionblur is. That is, the light emission period and the magnitude of motionblur are almost proportional to each other. On the other hand,concerning flicker synchronous with a frame, the longer the lightemission period is, the smaller the flicker observed. The shorter thelight emission period is, the larger the observed flicker. That is, thelight emission period and flicker have a trade-off relationship.

A solution to the two problems is multiplying the frame frequency by N.In many case, N=2. That is, the rate is doubled. When the framefrequency is doubled, the light emission period in each double-rateframe is halved. This also almost halves the motion blur. Regardingflicker as well, if an initial frame frequency of 60 Hz is doubled to120 Hz, the frequency of flicker falls outside the responsecharacteristic of human eyes. Hence, no flicker is observed.

As described above, doubling the frame frequency (more generallyspeaking, multiplying the frame frequency by N) has a large effect butposes a new problem.

For example, when the frame frequency of an original image signal is 60Hz, the image information is updated every 1/60 sec. If the framefrequency is doubled to display image data at 120 Hz, necessary imageinformation is missing every other frame. As a measure, identical imagesare displayed, for example, twice if the frame frequency is doubled.This solves flicker but cannot improve motion blur in the originalimage. In an impulse-type display device, doubled images are observed inpursuit (this phenomenon will be referred to as “double-blurring”hereinafter).

To reduce the motion blur or double-blurring and prevent flicker, twomethods are mainly used to double the frame frequency.

The first method detects the motion of an object in an original imageand estimates images between two frames. This is generally called anintermediate image generation method by motion compensation. In thisfirst method, an estimation error occurs under a specific condition. Inaddition, the amount of computation required is extremely high.

In the second method, a filter process is first performed for each frameof an input image to separate a spatial high-frequency componentstrongly related to motion blur and a spatial low-frequency componentstrongly related to flicker. The spatial high-frequency component isconcentrated in one sub-frame (one of the two double-rate framescorresponding to the original frame). The spatial low-frequencycomponent is distributed to both sub-frames (both of the two double-rateframes corresponding to the original frame).

In this specification, this second method will be called a “method ofseparating an image into spatial frequencies and distributing them tosub-frames for display”.

As the “method of separating an image into spatial frequencies anddistributing them to sub-frames for display”, Japanese Patent Laid-OpenNo. 6-70288 (to be referred to as patent reference 1 hereinafter),Japanese Patent Laid-Open No. 2002-351382 (to be referred to as patentreference 2 hereinafter), and U.S. Pre-Grant Publication No.2006/0227249A1 (to be referred to as patent reference 3 hereinafter) areknown.

SUMMARY OF THE INVENTION

The present invention can further improve the latter (second method)with a relatively light processing load.

In examining the “method of separating an image into spatial frequenciesand distributing them to sub-frames for display”, two problems areposed.

The first problem is that a display image in actual pursuit hasdistortion. The second problem is that it is impossible to make full useof the dynamic range of a display device.

The first problem is supposed to be caused when a spatial high-frequencycomponent and a spatial low-frequency component generated from eachframe of an input image are distributed to sub-frames, and the temporalcenters of gravity of their components shift in display.

In pursuit, the display time of each image corresponds to the spatialposition of the image observed in pursuit. If the temporal center ofgravity shifts, the spatial center of gravity shifts in the imageobserved in pursuit. Even in an image observed in pursuit, if thespatial center of gravity of the spatial high-frequency component andthat of the spatial low-frequency component have a shift relative to oneanother, an image having distortion such as ghosting or tail-blurring isobtained, as in a normal image.

In patent reference 1, the rate of an original image is doubled, and thespatial high-frequency component of one sub-frame (one of the twocorresponding double-rate frames) is limited, and the presence/absenceof the limitation is controlled in accordance with the motion of theimage. Hence, when the high-frequency component is limited, its temporalcenter of gravity shifts to the sub-frame in which the high-frequencycomponent is not limited. As a result, an image in which the temporalcenter of gravity of the high-frequency component and that of thelow-frequency component of the input image have a shift is displayed,and ghosting or tail-blurring may be observed in pursuit.

In patent reference 2, the rate of an original image is doubled, and thespatial high-frequency components of two double-rate frames (twosub-frames corresponding the frame of the input image) are increased ordecreased in accordance with the motion. In this case as well, thetemporal center of gravity of the spatial high-frequency componentshifts to the sub-frame in which the high-frequency component isincreased. Hence, ghosting or tail-blurring may be observed in pursuit,like the above-described case.

In patent reference 3, the rate of an original image is doubled, and thespatial high-frequency component of one sub-frame is increased, and thatof the other sub-frame is decreased by the same amount. In this case aswell, the temporal center of gravity of the spatial high-frequencycomponent shifts to the sub-frame in which the high-frequency componentis increased. Hence, ghosting or tail-blurring may be observed inpursuit, like the above-described case.

The second problem, that it is impossible to make full use of thedynamic range of a display device, will be described below.

When the display level of one sub-frame is moved to the other sub-frame,the maximum display intensity of the sub-frame whose display level hasrisen is limited. At this time, the display intensity of the sub-framewith the lower display level has not reached the maximum value. For thisreason, it is impossible to use the dynamic ranges of the two sub-framesat maximum.

The present invention has been made in consideration of theabove-described problems, and has as its object to provide a techniqueof mitigating at least the first and/or second problems using a simplearrangement.

For example, the present invention to solve the first problem has thefollowing arrangement.

According to one aspect of the present invention, there is provided animage processing apparatus configured to output moving image data whichis input at m frames per unit time as output moving image data at N×m(N≧2) frames per unit time, comprises:

a filter unit configured to separate image data from an input frame ofinterest into spatial high-frequency component image data and spatiallow-frequency component image data;

a storage unit configured to store the spatial high-frequency andspatial low-frequency component image data obtained by the filter unit;

a read unit configured to read the spatial high-frequency componentimage data and the spatial low-frequency component image data from thestorage unit N times;

a multiplication unit configured to, multiply the spatial high-frequencycomponent image data and the spatial low-frequency component image databy predetermined multiplier coefficients which are set to make atemporal center of gravity of the spatial high-frequency component imagedata match that of the spatial low-frequency component image data andmake temporal distribution of the spatial high-frequency component imagedata smaller than that of the spatial low-frequency component imagedata;

an addition unit (3) configured to add the spatial high-frequencycomponent image data and the spatial low-frequency component image dataafter multiplication by the multiplication unit every time the data isread; and

an output unit configured to output the summed image data from theaddition unit.

According to another aspect of the present invention, there is providedan image processing apparatus configured to receive input moving imagedata at m frames per unit time and output moving image data at 2m framesper unit time, comprises:

a receiving unit configured to receive input image data of each frame;

a filter unit configured to separate image data of an input frame ofinterest into spatial high-frequency component image data and spatiallow-frequency component image data;

a storage unit configured to store the input image data, the spatialhigh-frequency component image data, and the spatial low-frequencycomponent image data;

a calculation unit configured to calculate low-frequency averaged imagedata that is an averaged value of low-frequency component image data ofan input frame temporally adjacent the frame of interest and thelow-frequency component image data of the frame of interest stored inthe storage unit;

a generation unit configured to generate high-frequency emphasized imagedata based on the low-frequency component image data and thehigh-frequency component image data; and

an output unit configured to output, sequentially for each frame, thehigh-frequency emphasized image data and the low-frequency averagedimage data obtained by the calculation unit.

According to still another aspect of the present invention, there isprovided an image processing apparatus configured to output moving imagedata which is input at m frames per unit time as output moving imagedata containing N×m (N≧2) frames per unit time, comprises:

filter unit configured to separate input moving image data of a frame ofinterest into R (R≧2) spatial frequency band component images;

storage unit configured to store the R image data components obtained bythe filter unit;

reading unit configured to read the R image data components from thestorage unit N times;

multiplication unit configured to multiply the R image data componentsby predetermined multiplier coefficients which are set to make thetemporal centers of gravity of the R image data components match, andmake the temporal distribution of the R image data components smaller inascending order of the spatial frequency of the component image;

addition unit configured to adds the R spatial frequency band image datacomponents after multiplication by the multiplication unit every timethe reading reads the R spatial frequency band image data components;and

output unit which outputs the summed image data from the addition unit.

According to yet another aspect of the present invention, there isprovided an image processing apparatus configured to output moving imagedata which is input at m frames per unit time as output moving imagedata at N×m (N≧2) frames per unit time, comprises:

filter unit configured to separate an image of a frame of interest intoa spatial high-frequency image data component and a spatiallow-frequency image data component;

storage unit configured to store each spatial frequency image datacomponent obtained by the filter unit;

reading unit configured to read the spatial high-frequency image datacomponent and the spatial low-frequency image data component from thestorage unit N times;

multiplication unit configured to multiply the spatial high-frequencyimage data component and the spatial low-frequency image data componentby predetermined multiplier coefficients which are set to allocate thelow-frequency image data component to a larger number of frames than thehigh-frequency image data component;

addition unit which adds the spatial high-frequency image data componentand the spatial low-frequency image data component after multiplicationby the multiplication unit every time the reading unit reads out thespatial high-frequency image data component and the spatiallow-frequency image data component; and

output unit configured to output the summed image data from the additionunit.

According to still yet another aspect of the present invention, there isprovided a method of controlling an image processing apparatus whichoutputs moving image data input at m frames per unit time as outputmoving image data at N×m (N≧2) frames per unit time, the methodcomprises:

separating an image of a frame of interest into a spatial high-frequencyimage data component and a spatial low-frequency image data component;

storing each spatial frequency image data component obtained in thefiltering step in a storage unit;

reading the spatial high-frequency image data component and the spatiallow-frequency image data component from the storage unit N times;

multiplying the spatial high-frequency image data component and thespatial low-frequency image data component by predetermined multipliercoefficients which are set to make a temporal center of gravity of thespatial high-frequency image data component match that of the spatiallow-frequency image data component and make the temporal distribution ofthe spatial high-frequency image data component smaller than that of thespatial low-frequency image data component;

adding the spatial high-frequency image data component and the spatiallow-frequency image data component after multiplication every time thespatial high-frequency image data component and the spatiallow-frequency image data component are read; and

outputting the image data summed in the adding step.

According to yet still another aspect of the present invention, there isprovided a method of controlling an image processing apparatus whichinputs moving image data containing m frames per unit time and outputsmoving image data containing 2m frames per unit time, the methodcomprises:

inputting image data of each frame;

separating image data of an input frame of interest into a spatialhigh-frequency image data component and a spatial low-frequency imagedata component;

storing a low-frequency image data component of an immediately precedingframe of the frame of interest in a storage unit;

calculating a low-frequency image data average that is an averaged valueof the low-frequency image data component of a frame input next to theframe of interest obtained in the filtering step and the low-frequencyimage data component stored in the storage unit; and

sequentially outputting for each frame, high-frequency emphasized imagedata generated based on the low-frequency image data component and thehigh-frequency image data component, and the low-frequency image dataaverage.

According to still yet another aspect of the present invention, there isprovided a method of controlling an image processing apparatus whichoutputs moving image data input at m frames per unit time as outputmoving image data at N×m (N≧2) frames per unit time, the methodcomprises:

separating the input moving image data of a frame of interest into R(R≧2) spatial frequency band component images;

storing the R image data components obtained in a storage unit;

reading the R image data component from the storage unit N times;

multiplying the R image data components by predetermined multipliercoefficients which are set to make the temporal centers of gravity ofthe R image data components match, and make the temporal distribution ofthe R image data components smaller in ascending order of the spatialfrequency of the component image;

adding the R spatial frequency band image data components aftermultiplication every time the R spatial frequency band image datacomponents are read; and

outputting the summed image data from the addition.

According to yet still another aspect of the present invention, there isprovided a method of controlling an image processing apparatus whichoutputs moving image data input at m frames per unit time as outputmoving image data containing N×m (N≧2) frames per unit time, the methodcomprises:

separating an input image of a frame of interest into a spatialhigh-frequency image data component and a spatial low-frequency imagedata component;

storing each spatial frequency image data component obtained in astorage unit;

reading the spatial high-frequency image data component and the spatiallow-frequency image data component from the storage unit N times;

multiplying the spatial high-frequency image data component and thespatial low-frequency image data component by predetermined multipliercoefficients which are set to allocate the low-frequency image datacomponent to a larger number of frames than the high-frequency imagedata component;

adding the spatial high-frequency image data component and the spatiallow-frequency image data component after multiplication every time thespatial high-frequency image data component and the spatiallow-frequency image data component are read; and

outputting the image data summed in the addition.

According to the present invention, it is possible to, for example,reduce motion blur in a hold-type display device and reduce flicker inan impulse-type display device using a simple process. It is alsopossible to suppress distortion such as ghosting or tail-blurring inpursuit. The invention also enables prevention of any adverse effects onimage quality caused by a decrease in brightness or saturation of thelevel of one sub-frame.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments (with reference to theattached drawings).

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing an arrangement of an image processingapparatus according to the first embodiment;

FIG. 2 is a block diagram showing another arrangement of the imageprocessing apparatus according to the first embodiment;

FIG. 3 is a view showing examples of image signal waveforms along a timeaxis in input frames and double-rate frames according to the firstembodiment;

FIG. 4 is a block diagram showing the arrangement of a conventionalimage processing apparatus;

FIG. 5 is a block diagram showing the arrangement of anotherconventional image processing apparatus;

FIG. 6 is a block diagram showing the arrangement of yet anotherconventional image processing apparatus;

FIGS. 7A to 7D are graphs for explaining the causes of tail-blurring;

FIGS. 8A to 8C are graphs for explaining why motion blur does not occurin an image without a change over time;

FIGS. 9A to 9C are graphs showing image signals SH and SL assuming thatpursuit is done for the images shown in FIGS. 8A to 8C;

FIGS. 10A to 10C are graphs showing image signals SH and SL assumingthat pursuit is done in the first embodiment;

FIGS. 11A to 11C are graphs showing pursuit waveforms of a prior artarrangement;

FIGS. 12A to 12C are graphs showing pursuit waveforms according to thefirst embodiment;

FIG. 13 is a timing chart showing the transition of brightness ofsub-frames SH and SL in a hold-type display device according to thesecond embodiment;

FIG. 14 is a timing chart showing the transition of brightness ofsub-frames SH and SL in an impulse-type display device according to thesecond embodiment;

FIG. 15 is a timing chart showing the frame display timing of aconventional hold-type display device;

FIG. 16 is a timing chart showing the frame display timing of aconventional impulse-type display device;

FIG. 17 is a block diagram showing an arrangement of an image processingapparatus according to the second embodiment;

FIG. 18 is a block diagram showing another arrangement of the imageprocessing apparatus according to the second embodiment;

FIG. 19 is a block diagram of an adaptive multi-stage (4-stage) LPFaccording to the second embodiment;

FIG. 20 is a block diagram of an adaptive 2-stage LPF according to thesecond embodiment;

FIG. 21 is a timing chart for explaining process contents according tothe third embodiment;

FIG. 22 is a timing chart for explaining process contents according tothe third embodiment;

FIG. 23 is a timing chart for explaining preferable process contentsaccording to the third embodiment;

FIG. 24 is a timing chart for explaining preferable process contentsaccording to the third embodiment;

FIG. 25 is a view showing gain coefficients in each band of fivesub-frames in 5×-rate playback according to the fourth embodiment;

FIG. 26 is a block diagram showing the arrangement of an imageprocessing apparatus according to the fourth embodiment;

FIG. 27 is a flowchart illustrating the process procedure according to amodification of the first embodiment;

FIG. 28 is a table showing a frame relationship according to the firstembodiment;

FIG. 29 is a view showing the distribution ratio for sub-framesaccording to the first embodiment;

FIG. 30 is a block diagram showing the arrangement of an imageprocessing apparatus according to a modification of the firstembodiment;

FIG. 31 is a block diagram of an adaptive multi-stage (4-stage) LPFaccording to the second embodiment, which performs the filtering processfor an entire image; and

FIG. 32 is a block diagram of an adaptive 2-stage LPF according to thesecond embodiment, which performs the filtering process for an entireimage.

DESCRIPTION OF THE EMBODIMENTS

The embodiments of the present invention will now be described withreference to the accompanying drawings.

For easy understanding of the embodiments of the present invention, therelevant prior art will be described first with reference to FIGS. 4 to6.

In the prior art arrangement shown in FIG. 4, input field images areselectively temporarily saved in two field memories and alternatelyoutput at double the input rate via a switch SW0, thereby doubling theframe rate of the original signal. At this time, the high-frequencycomponent of the spatial frequency of one sub-frame is suppressed. As aresult, the sub-frame (expressed by SL in FIG. 4) contains a relativelysmall amount of spatial high-frequency component. The other sub-frame(expressed by SH in FIG. 4) contains a relatively large amount ofspatial high-frequency component. The spatial high-frequency componentis therefore localized in one of the sub-frames of the output image.This reduces motion blur.

In another prior art arrangement shown in FIG. 5, a frame converterdoubles the frame rate of an input image. A filter LPF/HPF separates thespatial frequency into a spatial low-frequency component, “Low”, and aspatial high-frequency component, “High”. The high-frequency componentHigh is multiplied by a predetermined gain α for each double-rate frame(or for one of the sub-frames viewed from the input image frame). Thepolarity of α is changed in every double-rate frame by using α having apositive value in one of the double-rate frames and α having a negativevalue in the other double-rate frame. If it is determined that themotion of the image is large, the absolute value of α may be increased.The spatial high-frequency component is therefore localized in onedouble-rate frame (or one sub-frame viewed from the input image) SH.This reduces motion blur.

In still another prior art arrangement shown in FIG. 6, an input imageA[i] passes through a filter HPF to generate spatial high-frequencycomponent data H[i]. The spatial high-frequency component data H[i] isadded to the input image A[i] to generate high-frequency emphasizedimage data SH[i]. The spatial high-frequency component data H[i] issubtracted from the input image A[i] to generate low-frequency imagedata SL[i]. These data are switched at a frequency twice the framefrequency of the input image by a switch SW0, thereby outputting adouble-rate image in which the spatial high-frequency component isconcentrated to one of the double-rate frames (one sub-frame viewed fromthe input image frame). This reduces motion blur.

In these prior art arrangements, the two corresponding double-rateframes (or one of the sub-frames viewed from the input image), that is,the frames SH and SL, are generated on the basis of one input frame.When, for example, the frame SH is displayed first, and the frame SL isthen displayed, the frame SH contains a large amount of spatialhigh-frequency component of the input image frame. Hence, the spatialhigh-frequency component shifts forward on a time basis. On the otherhand, the spatial low-frequency component is distributed to both theframes SH and SL and does not shift at all on a time basis. Hence, thespatial high-frequency component shifts forward on a time basis relativeto the spatial low-frequency component.

When pursuit is done for a moving portion of a moving image, the forwardshift on a time basis is equivalent to spatial shift in the movingdirection in an image observed in pursuit. Hence, the spatialhigh-frequency component in the image observed in pursuit shifts in themoving direction relative to the spatial low-frequency component. Inthis case, an image having ghosting or tail-blurring is observed.

First Embodiment

The first embodiment can mitigate the first problem described above.More specifically, the first embodiment can enable the elimination ofthe relative time shift in display (output) of spatial high-frequencycomponent image data and spatial low-frequency component image data. Inthe first embodiment, it is therefore possible to prevent or reduceghosting or tail-blurring in an image observed in pursuit.

FIG. 1 is a block diagram showing an arrangement of an image processingapparatus according to the first embodiment.

FIG. 1 shows an apparatus which receives each frame of moving image dataand generates two sub-frames (double-rate moving image data) from theinput moving image data of one frame. The input moving image data of oneframe is represented by A[i], and spatial high-frequency component dataof the input moving image data A[i] is represented by H[i] (spatialhigh-frequency data). Spatial low-frequency component data (spatiallow-frequency component image data) is represented by L[i]. One of thedouble-rate frames (or sub-frames) which are alternately output torealize a double-rate image is represented by SH[i], and the other isrepresented by SL[i]. The “[i]” index notation indicates the ith frameof the input moving image.

A lowpass filter 1 (to be referred to as an LPF 1 hereinafter) is atwo-dimensional lowpass filter. This filter does not define a specificfunction. For example, a Gaussian function is usable. Alternatively, amoving average or a weighted moving average is usable. In the followingdescription, an effective distance that is ½ the maximum value of afilter coefficient is defined as “distance constant value: d”. Thedistance constant value d indicates a wavelength corresponding to aspatial frequency that is equal to a cutoff frequency in limiting thebandwidth of an image by a spatial frequency filter. The unit of d is“pixel”.

The LPF 1 first cuts off (filters), from the input image A[i], upperspatial frequencies above a predetermined constant value, therebygenerating the spatial low-frequency component data L[i]. A subtractor 2calculates the spatial high-frequency component data H[i] by subtractingL[i] from the original image data A[i] in accordance withH[i]=A[i]−L[i]

An adder 3 adds the high-frequency component data H[i] to the originalimage data A[i], thereby generating sub-image data (spatialhigh-frequency emphasized image data) SH[i] containing a large amount ofspatial high-frequency components.SH[i]=A[i]+H[i]=L[i]+2×H[i]

Each of frame delay circuits 4 and 5 delays the current frame by oneframe to perform calculation between the current frame and the nextframe. The frame delay circuit 4 eliminates the frame shift between asignal input to a terminal a of a switch 8 and a signal input to aterminal b.

An adder 6 adds spatial low-frequency component data L[i−1] generated bythe LPF 1 and delayed by one frame to L[i] of the next frame.

A divider 7 calculates an average value by dividing the image data fromthe adder 6 by “2” and outputs it as low-frequency averaged image dataSL[i−1].SL[i−1]={L[i−1]+L[i]}/2

The frame delay circuit 4 delays SH[i] by one frame to generate SH[i−1]so that SH[i] calculated by the adder 3 matches SL[i−1].

The thus generated SH[i−1] and SL[i−1] are selected by the switch 8 at afrequency twice that of the input image frames, and are sequentiallyoutput. This enables output of a double-rate image signal containing aspatial high-frequency component localized in one frame. It is thereforepossible to realize an image with small motion blur and small flicker.

In the embodiments of the present invention, calculations anddefinitions are explained for displayed brightness (displayed lightintensity) as an example. In the embodiments, a timing chart showing,for example, the waveform of an image expresses brightness along theordinate. Hence, the present invention is most effective when it isapplied to image data defined as data proportional to displayedbrightness (displayed light intensity). However, the present inventionis not always limited to such a case. The present invention is alsoapplicable to image data in a range generally used (image data havingslight nonlinearity between data values and displayed brightness orlight intensity). Even in this case, the concept of the presentinvention approximately holds and provides a sufficiently advantageouseffect.

FIG. 1 shows an example using the SH[i−1]. FIG. 2 shows an example inwhich a circuit arrangement equivalent to that in FIG. 1 is formed usinga high pass filter (HPF).

Reference numeral 11 in FIG. 2 denotes a high pass filter (to bereferred to as an HPF hereinafter). FIG. 2 shows a circuit arrangementwhich assumes the LPF 1 in FIG. 1 and the HPF 11 in FIG. 2 have arelationship given by A[i]=H[i]+L[i]. Reference numeral 13 in FIG. 2denotes a subtractor; 12 and 16, adders; 14 and 15, frame delaycircuits; 17, a divider; and 18, a switch. The constituent elements cansufficiently be understood based on the illustrated arrangement, and adetailed description thereof will be omitted.

FIG. 3 shows the correspondence relationship between the input imageA[i] and the output images SH[i] and SL[i] in the first embodiment. Thesequence of frames proceeds with timefrom the top to the bottom of theFigure. An output image is output with a time delay of at least oneframe with respect to a corresponding input image.

The output images SH[i−1], SH[i], and SH[i+1] corresponds to inputimages A[i−1], A[i], and A[i+1], respectively. The elements of theoutput images are calculated from the corresponding frames of the inputimages. On the other hand, for example, constituent elements L[i] andL[i+1] of an output image SL[i] are calculated from A[i] and A[i+1],respectively. SL[i] is the average value of the elements.

With this relationship, ½ of L[i] that is the spatial low-frequencycomponent data of A[i] is output (displayed) as SH[i], and the remaining¼ components are displayed as SL[i−1] and SL[i].

Hence, the temporal center of gravity of display of L[i] is located atthe intermediate point between SL[i−1] and SL[i], that is, at the centerof the display period of SH[i]. On the other hand, H[i] that iscalculated as the spatial high-frequency component of A[i] is whollydisplayed as SH[i]. The temporal center of gravity of display of H[i] islocated at the center of the display period of SH[i]. Consequently, thetemporal center of gravity of display of L[i] matches that of H[i]. Thatis, distortion such as ghosting or tail-blurring in an image observed inpursuit can be prevented.

<Order of SH and SL>

This embodiment will be explained assuming that the two sub-framesignals SH[i] and SL[i] corresponding to one input data frame A[i] areoutput such that they are displayed in that order. For example, FIG. 28is a table that explicitly shows this order. However, the presentinvention is not limited to this order, and the order may be reversed.Even in the reversed order, the present invention has the same effect.

The order that outputs SL later, as in the explanation of thisembodiment, will be referred to as a “forward order”, and the order thatoutputs SL first will be referred to as a “reverse order” hereinafter.

In the “forward order”, SL takes an average value by referring to thelow-frequency component of a frame of interest and that of animmediately succeeding frame. Actually, after waiting one frame for theimmediately succeeding frame to be input, the average value between thetemporarily saved current frame and the newly input frame is calculated.Hence, the output low-frequency image is expressed bySL[i−1]=(L[i−1]+L[i])/2

In contrast, in the “reverse order”, SL takes an average value byreferring to the low-frequency component of a frame of interest and thatof an immediately preceding frame. Hence, the output low-frequency imageis expressed bySL[i]=(L[i]+L[i−1])/2

The two equations have the same form except that the output framenumbers differ by one.

Anyway, one low-frequency image data SL corresponds to the average valueof low-frequency images reflected on two sub-frames SH adjacent to thesub-frame of interest.

As described in association with the prior art, bilaterally asymmetricaldistortion occurs in a pursuit image, as shown in FIGS. 9A to 9C or 11Ato 11C. The shape of distortion changes between the “forward order” andthe “reverse order”. More specifically, the shape is reversed in thelateral direction. Especially, tail-blurring distortion reverses itsdirection.

Detailed Explanation of Operation of First Embodiment

In this embodiment, N=2, and the spatial frequency has two bands, a highfrequency H and a low frequency L. In the present invention, however,the frame frequency of an input image may be multiplied by N (or dividedinto N sub-frames), and the input image may be divided into a pluralityof spatial frequency bands. In this case, of the plurality of bands, ahigher-frequency component is concentrated on the time basis, whereas alower-frequency component is distributed on the time basis.Additionally, the temporal centers of gravity are arranged to match.Hence, N may be 2 or more so that an input image is divided into aplurality of spatial frequency bands.

To satisfy the above conditions of the spatial high-frequency componentH and spatial low-frequency component L, and concentrate H to onesub-frame, the number of sub-frames to distribute must be odd and atleast 3. In this embodiment, N=2, and display is done using sub-framesobtained by doubling the frame rate. One input image is distributed tothree sub-frames. That is, the number Nd of sub-frames to distribute is3. In this embodiment, the relationships (N=2, and Nd=3) aresimultaneously achieved as frames adjacent to each other in an inputimage share one of the sub-frames at the ends as the sub-frame todistribute. If N is an even number, the components are distributed to Nd(Nd≧N) that is an odd number, and adjacent frames share, for example,one sub-frame. The coefficient of the sub-frame is obtained by addingthe original distribution coefficients. This solves the mismatch betweenN and Nd. The sum value must not exceed the coefficient of anothersub-frame of the same spatial frequency component.

FIG. 28 is a table showing the relative relationship of images. That is,FIG. 28 faithfully expresses the operation in FIG. 1 in accordance withthe transition of frames. The transition of frames is shown along thehorizontal direction of FIG. 28. The types of images or image componentsare shown along the vertical direction. Each cell of the first linerepresents one frame of an input image sequentially from the left. Fromthe second line, each column represents one sub-frame sequentially fromthe left. An output image is delayed by one frame from an input image A.Hence, in FIG. 28 as well, the value i in the second line is smaller byone than that in the first line.

The first line of FIG. 28 represents the input image A. The second linerepresents sub-frames to be output. The third to seventh lines show thespatial high-frequency component H and the spatial low-frequencycomponent L which are extracted from each frame of the input image Ausing a filter and multiplied by coefficients. The eighth linerepresents the breakdown of the sum output components of the sub-framesin the second line.

Frequency components corresponding to A[i] in the first line of FIG. 28are shown in the fourth to sixth columns of the fifth line. FIG. 29shows details of this portion. In this way, each frame of the inputimage is distributed (multiplied) to three sub-frames using thecoefficients shown in FIG. 28. In FIG. 29, GL1 to GL3 are {0.5, 1, 0.5},and GH1 to GH3 are {0, 2, 0}.

FIG. 30 shows a modification of the first embodiment. The arrangement inFIG. 30 has the same function as in FIG. 1. FIG. 1 illustrates theembodiment in the simplest form. FIG. 30 shows, for example, how todistribute the input image A to three sub-frames at a predeterminedratio in accordance with FIG. 28. It is also possible to implement anarrangement example different from FIG. 1 by appropriately changing thecoefficients of multipliers in FIG. 30, while satisfying a predeterminedcondition.

The block diagram in FIG. 30 will be described. A doubling circuit 50doubles an input frame sync signal (e.g., 60 Hz) to generate a syncsignal clk2 of 120 Hz. A filter 53 separates the input image into aspatial high-frequency component (H) and a spatial low-frequencycomponent (L) and corresponds to a combination of the LPF 1 and thesubtractor 2 in FIG. 1. Each of switches 51 and 52 selects the outputterminals in the order of (0)→(1) or (1)→(0) every time the frame syncsignal of A[i] is input.

Each of switches 58 to 61 selects its four output terminals in the orderof (1)→(2)→(3)→(4)→(1) . . . step by step every time the output clk2from the doubling circuit 50 is input. The switches 58 and 59 areconnected to channels of the same number. The switches 60 and 61 arealso connected to channels of the same number. However, the switches 60and 61 are connected to channels whose number differs by two steps fromthat of the switches 58 and 59. The output terminals (4) of the switchesare dummy connections to generate timings. For the operation shown inFIG. 28, each switch selects the output terminals in the order of(1)→(2)→(3)→(4)→(1)→(2) . . . in accordance with clk2.

A multiplier having a predetermined coefficient is connected to eachchannel of the switches 58 to 61. Each of adders 62, 63, 64, and 65 addsthe outputs from the corresponding switch. An adder 66 adds the outputsfrom the adders 62 to 65.

The switches 58 and 60 have the same combination of coefficients ofmultipliers. The switches 59 and 61 also have the same combination ofcoefficients of multipliers.

The coefficients GH1 to GH3 of the multipliers of the switches 58 and 60are related to a spatial high-frequency component and indicated by the Hcomponent in FIG. 29. The coefficients GL1 to GL3 of the multipliers ofthe switches 59 and 61 are related to a spatial low-frequency componentand indicated by the L component in FIG. 29.

The flow of signals in FIG. 30 will now be described. FIG. 30 shows thearrangement at the timing of the fourth column in FIG. 28 (i.e., atiming corresponding to the sub-frame SL[i−1]). The filter 53 separatesthe input signal A[i] into the spatial high-frequency component H[i] andthe spatial low-frequency component L[i]. The separated spatialhigh-frequency component H[i] and spatial low-frequency component L[i]are sent to the switches 51 and 52, respectively. The switches selectthe output terminals (1) to temporarily store the frequency componentsin frame memories 56 and 57.

The output terminals (0) of the switches 51 and 52 previously output thecomponents from the immediately preceding frames H[i−1] and L[i−1] toframe memories 54 and 55 so that they temporarily store the signals.

The frame memories 54, 55, 56, and 57 receive new data from the filter53 and update their contents every time the switches 51 and 52 areconnected to corresponding terminals. Each stored data item is held fora period corresponding to two frame periods in the input image.Additionally, immediately before connections of the switches 58 to 61change from the output terminal (4) to the output terminal (1), thecontents of the corresponding frame memories are updated.

In this state, the switches 58 to 61 sequentially change the outputterminals. These outputs are added to obtain the output of thisembodiment. For example, when the switches 58 and 59 change the outputterminals in the order of (1)→(2)→(3), it indicates that componentscorresponding to A[i−1] in FIG. 28 are output, that is, the spatialfrequency band components are output in the order of second column→thirdcolumn→fourth column of the fourth line. Similarly, when the switches 60and 61 change the output terminals in the order of (1)→(2)→(3), itindicates that components corresponding to A[i] in FIG. 28 are output,that is, the spatial frequency band components are output in the orderof fourth column→fifth column→sixth column of the fifth line.

The output signal of this block diagram obtained in the above-describedway corresponds to the eighth line (lowermost stage) of FIG. 28. Thecoefficients shown in FIG. 29 are used as the coefficients GL1 to GL3and GH1 to GH3 of the multipliers shown in FIG. 30. As long as thesecoefficients are used, the arrangement in FIG. 30 has the same functionas in FIG. 1. To satisfy the feature of the present invention, it isnecessary that, for example, the first and third terms are equal inassociation with GL1 to GL3, the sum of the coefficients is 2, and thedouble value of the coefficient at each end is less than or equal tothat at the center. Hence, GL1 to GL3 can be, for example, {0.4, 1.2,0.4} or {0.3, 1.4, 0.3}. When GL1 to GL3 in FIG. 30 are replaced withthese coefficients, slightly more flicker is generated as compared tothe first embodiment, though an embodiment with smaller distortion canbe implemented. The latter coefficients tend to provide a larger effect.

As a result, an image having smaller motion blur and smaller flicker canbe obtained, as in the first embodiment.

<Cause of Tail-Blurring>

The reason why ghosting or tail-blurring is generated when the temporalcenter of gravity shifts between L[i] and H[i] will be described below.An example will be explained here in which the frame frequency of theinput image is 60 Hz.

Note that tail-blurring is a kind of distortion that draws a trailbefore or after an edge of an object in pursuit.

Consider pursuing an image which has a brightness scanning line with arectangular waveform, and which moves V (pixels) per 1/60 sec, as shownin FIG. 7A. That is, the image is a moving image having an object whichmoves V (pixels) between frames.

In a hold-type display device having a pulse width of, for example, 1/60sec (frame frequency: 60 Hz), motion blur with a width of V (pixels) isobserved (corresponding to the tilted portions shown in FIG. 7B).

In a hold-type display device having a pulse width of, for example,1/120 sec (frame frequency: 120 Hz), motion blur with a width of V/2(pixels) is observed (corresponding to the tilted portions shown in FIG.7C).

In impulse driving, the pulse width is small. Hence, a pursuit imagerelatively close to the waveform of a still image is obtained. FIG. 7Dshows a case in which the pulse width is 1/1200 sec. In any case, motionblur represented by the width of a tilted portion has a value almostequal to the pulse width. The frame frequency is not directly related tothis value. In a hold-type display device, the pulse width equals oralmost equals “1/frame frequency”.

FIGS. 8A to 8C show waveforms obtained by separating the input image Ainto a spatial high-frequency component and a spatial low-frequencycomponent using an HPF or LPF. FIG. 8A shows the waveform of the inputimage A. SH (=L+2H) and SL (=L) have waveforms shown in FIGS. 8B and 8C.However, FIGS. 8A to 8C show not the waveforms of an image observed inpursuit but those of a still image. Hence, there is no influence ofmotion blur.

FIGS. 9A to 9C are views assuming pursuit of SH[i] and SL[i] in FIGS. 8Band 8C.

Since the display period is 1/120 sec, the waveform exhibits motion blurcorresponding to the pulse width of 1/120 sec shown in FIG. 7C. In priorart systems, since SH[i] is generated from the same frame as that ofSL[i], they are frames of the same time. In pursuit, SL[i] is formedwith a shift with respect to SH[i] in a direction reverse to the movingdirection of the image by ½ (i.e., V/2) of the moving distance in aframe period.

Actually, SH[i] and SL[i] in FIGS. 9A and 9B are alternately displayedevery 1/120 sec. Hence, the actual human vision recognizes the compositewaveform in FIG. 9C. This waveform has tail-blurring as compared to thatin FIG. 7C. The first embodiment reduces this distortion.

In this embodiment, for example, SL[i] which is displayed between SH[i]and SH[i+1] is formed from the average value of spatial low-frequencycomponents corresponding to them, that is, {L[i]+L[i+1]}/2, as shown inFIG. 3. FIGS. 10A to 10C show SH[i] (=L[i]+2×H[i]), SL[i](={L[i]+L[i+1]}/2), and the waveform of a pursuit image, as in FIGS. 9Ato 9C.

As described above, in pursuit, L[i] is generated from the same frame ofthe input image as that of SH[i] which is displayed 1/120 sec earlier,and therefore formed with a shift in the direction opposite the movingdirection of the image by ½ (V/2) of the moving distance per frame. Onthe other hand, L[i+l] is generated from the same frame of the inputimage as that of SH[i+l] which is displayed 1/120 sec later, andtherefore formed with a shift in the direction opposite the movingdirection of the image by ½ (V/2) of the moving distance per frame.Hence, the waveform SL[i] (={L[i]+L[i+1]}/2) obtained by adding them anddividing the sum by 2 is located almost the same position as SH[i] interms of temporal center of gravity, as shown in FIG. 10B.

Actually, SH[i] and SL[i] in FIGS. 10A and 10B are alternately displayedevery 1/120 sec. Hence, the actual human vision recognizes the compositewaveform (FIG. 10C). This waveform is bilaterally symmetrical and has notail-blurring as compared to that in FIG. 9C. Reducing the tail-blurringdistortion is an effect of this embodiment.

An example has been described above in which a hold-type display device,whose pulse width equals the frame period, displays an input image witha frame frequency of 60 Hz at a double rate. Even in an impulse-typedisplay device, the problems of the prior art and the effect of theembodiment are the same.

Assume that an impulse-type display device has a pulse width of 1/100 ofthe frame period, that is, the input image has a pulse width of 1/600sec (1.67 msec) and a frame frequency of 60 Hz. Consider a case in whichwhen the frame rate is doubled, the pulse width is ½ of it ( 1/1200 sec(=0.83 msec)).

In this case, the pursuit image of the prior art has waveforms as shownin FIGS. 11A to 11C, and the pursuit image of this embodiment haswaveforms as shown in FIGS. 12A to 12C. FIGS. 11A to 11C correspond toFIGS. 9A to 9C. FIGS. 12A to 12C correspond to FIGS. 10A to 10C.

V/2 in FIGS. 9A and 10A is equivalent to a distance corresponding to thepulse width that is the cause of motion blur and is therefore replacedwith V/20 in FIGS. 11A and 12A. V/2 in FIGS. 9B and 10B is equivalent toa distance corresponding to the time difference between frames andtherefore remains V/2 even in FIGS. 11B and 12B.

Modification of First Embodiment

An example will be described in which a process equivalent to the firstembodiment is implemented by a computer program. An apparatus forexecuting the computer program can be an information processingapparatus such as a personal computer (PC). The structure and featuresof the hardware comprising a PC are well known, so will not be describedhere. We assume that a moving image data file containing m frames perunit time is already stored in a storage device (or storage medium) suchas a hard disk. An example will be described in which the CPU executingthe application (computer program) of this modification converts thefile into moving image data to be played back at twice the frame rate,that is, 2 m frames per unit time, and saves the conversion result inthe hard disk as a file. The conversion target moving image data isstored in the storage device. The moving image data after double-rateconversion is also stored in the storage device. Hence, the applicationneed not display the double-rate conversion result. That is, note thatthe CPU need not execute the process in synchronism with the frame rateof the moving image represented by the conversion target moving imagedata, either. The application of the modification is also stored in thehard disk. The CPU loads the application to the RAM and executes it.

FIG. 27 is a flowchart illustrating the process procedure of theapplication. The execution process procedure of the CPU will bedescribed below with reference to FIG. 27. In the following explanation,memory areas A and B are areas allocated in the RAM.

In step S1, the CPU reads out image data A[i] of one frame from theconversion target moving image data to the RAM. If the data is encoded,the CPU executes a corresponding decoding process.

In step S2, the CPU filters the input image data frame of interest A[i]using a preset LPF to generate spatial low-frequency component dataL[i].

The process advances to step S3. The CPU generates spatialhigh-frequency component data H[i].H[i]=A[i]−L[i]

In step S4, the CPU generates high-frequency emphasized image data SH[i]and temporarily stores it in the RAM.SH[i]=A[i]+H[i]=L[i]+2×H[i]

In step S5, the CPU reads out an immediately preceding frame SH[i−1]from the memory area A as a current frame.

In step S6, the CPU saves SH[i] in the memory area A to prepare for theprocess of the next input frame.

In step S7, the CPU reads out an immediately preceding frame SL[i−1]from the memory area B as a current frame.

In step S8, the CPU saves SL[i] in the memory area B to prepare for theprocess of the next input frame.

In step S9, the CPU generates low-frequency averaged image data SL[i−1].SL[i−1]={L[i−1]+L[i]}/2

After generating the high-frequency emphasized image data SH andlow-frequency averaged image data SL, the CPU advances the process tostep S10. In step S10, the CPU outputs the two generated image data(sub-frames) as output moving image data.

SH[i−1] used in step S5 uses the process result in step S6 of thepreceding cycle. SL[i−1] used in step S7 uses the process result in stepS8 of the preceding cycle. In double-rate conversion of the first frameof the image data, no low-frequency image data of a preceding frameexists. In this case, the process is executed assuming an appropriatepixel value.

In step S11, the CPU determines whether all frames of the conversiontarget moving image data are converted. This process can be done bydetermining whether the file end of the conversion target moving imagedata is detected.

If NO in step S11, the variable i is incremented by “1”, and the processfrom step S1 is repeated.

If YES in step S11, the CPU finishes the series of double-rateconversion processes.

As described above, as compared with the first embodiment, theconversion process speed depends on the CPU. It is however possible tocreate a double-rate moving image data file having the same function andeffect as in the first embodiment.

Second Embodiment

In the second embodiment, an example of a solution to theabove-described second problem will be described. FIG. 17 is a blockdiagram showing an arrangement of an image processing apparatusaccording to the second embodiment. The same reference numerals as inFIG. 1 denote the same components in FIG. 17, and a description thereofwill be omitted.

The second problem occurs when each frame of an input image is filteredto separate a spatial high-frequency component related to motion blurand a spatial low-frequency component related to flicker. That is, in amethod of displaying an image by concentrating the spatialhigh-frequency component to one sub-frame and distributing the spatiallow-frequency component to both sub-frames, it is impossible to makefull use of the dynamic range of the display device.

“One sub-frame” indicates one of two double-rate frames corresponding tothe input image frame. “Both frames” indicates both of the twodouble-rate frames corresponding to the input image frame.

In such a display method, the spatial high-frequency component isconcentrated to one sub-frame, as in a hold type shown in FIG. 13 or animpulse type shown in FIG. 14. Hence, if, for example, an image withhigh brightness is displayed, saturation occurs first in a sub-framethat displays SH[i], and a sub-frame that displays SL[i] does not yetreach saturation even after the above-described sub-frame is saturated.For this reason, the maximum display brightness becomes smaller thanthat of a normal display method (FIGS. 15 and 16). To actually avoidsaturation, the original image signal must be multiplied by a ratiosmaller than 1. This ratio will be defined as k (0.5≦k≦1) hereinafter.In the first embodiment, an example in which saturation is not takeninto consideration has been described. This example can also be regardedas a case using k=1.

To prevent any decrease in brightness, k needs to be as close to “1” aspossible. To allow this, it is necessary to minimize the absolute valueof the spatial high-frequency component portions in FIG. 13 or 14 asmuch as possible to reduce the non-uniformity of brightness. The spatialhigh-frequency component portions in FIG. 13 or 14 can take either apositive value or a negative value. For the sake of simplicity, FIGS. 13and 14 show a case in which the value is positive.

For this purpose, a distance constant value d of the LPF or HPF isminimized. More specifically, the spatial frequency to separate thespatial low-frequency component and the spatial high-frequency componentneeds to be set high. In other words, the distance constant value of thefilter must be made small to decrease the absolute value of thehigh-frequency component.

If the distance constant value d is too small, an overshoot portionshown in FIG. 9C or 10C becomes notable. When the distance constantvalue d is sufficiently large, the waveform in FIG. 10C resembles thatin FIG. 7C. The overshoot becomes notable as the pursuit target imagemoves faster. That is, the minimum and necessary value d for reproducinga waveform needs to be determined based on the speed of the input imageas the pursuit target.

In the second embodiment, a large distance constant value d is set for aportion that moves fast in an input image. A small distance constantvalue d is set for a portion that moves slowly. That is, in the secondembodiment, the distance constant value d is determined adaptively foreach area of an image in consideration of the spatial frequencydistribution of the input image and the moving speed of each portion.

As shown in FIG. 17, an adaptive multi-stage low-pass filter 20 is usedin the second embodiment. FIG. 19 shows the detailed arrangement of theadaptive multi-stage low-pass filter, which includes a plurality of LPFshaving a stage structure.

FIG. 19 shows an example in which the number of stages=4, that is, fourlow-pass filters which execute the filter process of a current frame. Asshown in FIG. 19, four LPFs 23, 26, 29, and 31 are connected in a daisychain. The plurality of LPFs 23, 26, 29, and 31 function as a firstfilter unit. Similarly, for a preceding frame as well, three LPFs 22,24, and 28 are connected in a daisy chain. The plurality of LPFs 22, 24,and 28 function as a section filter unit. The LPFs 22 and 23 use thesame distance constant value. The LPFs 24 and 26 are also set to use thesame distance constant value. The LPFs 28 and 29 also use the samedistance constant value. The distance constant value of each low-passfilter LPFn (n=1, 2, 3, 4) is set to be do (n=1, 2, 3, 4) in accordancewith the numerical subscript. The larger the numerical subscript is, thelarger the set distance constant value is. That is, the upper-limitspatial frequency of the frequency range to be passed becomes lower inascending order of numerical subscript, that is, as the filter positiongoes downstream. Hence, the distance constant value can also be regardedas information that specifies the upper-limit spatial frequency forfiltering.

The functions of blocks in FIG. 19 will now be described.

A delay circuit (DL) 21 delays input image data by one frame and outputsit. In fact, the DL 21 can be formed from a FIFO memory for storing oneframe of image data. When the current frame of image data input to theadaptive multi-stage low-pass filter 20 is A[i], the DL 21 outputs thepreceding frame A[i−1] of the image data.

The LPFs 22 and 23 filter the input image data in accordance with adistance constant value d1.

The LPFs 24 and 26 receive, as input data, signals L1[i−1] and L1[i]from the upstream LPFs 22 and 23 and area data AREA1 output from adifference comparator 25. A distance constant value d2 is preset for theLPFs 24 and 26. The LPFs 24 and 26 respectively filter L1[i−1] and L1[i]corresponding to the pixels in the area represented by the area dataAREA1 in accordance with the distance constant value d2. Although thefiltering process is done in that area, the range where the image datais changed by this process is wider than the area by the distanceconstant value d2. The filtering results are output as L2[i−1] andL2[i].

The LPFs 28 and 29 receive, as input data, the signals L2[i−1] and L2[i]from the LPFs 24 and 26 and area data AREA2 output from a differencecomparator 27. A distance constant value d3 is preset for the LPFs 28and 29. The LPFs 28 and 29 respectively filter L2[i−1] and L2[i]corresponding to the pixels in the area represented by the area dataAREA2 in accordance with the distance constant value d3. Although thefiltering process is done in that area, the range where the image datais changed by this process is wider than the area by the distanceconstant value d3. The filtering results are output as L3[i−1] andL3[i].

The LPF 31 receives, as input data, the signal L3[i] from the LPF 29 andarea data AREA3 output from a difference comparator 30. A distanceconstant value d4 is preset for the LPF 31. The LPF 31 filters L3[i]corresponding to the pixels in the area represented by the area dataAREA3 in accordance with the distance constant value d4. Although thefiltering process is done in that area, the range where the image datais changed by this process is wider than the area by the distanceconstant value d4. The filtering result is output as a final result L[i]of the adaptive multi-stage low-pass filter 20.

The difference comparator 25 receives L1[i−1] and L1[i], calculates theabsolute value of the difference for each pixel, and based on theresult, calculates the area data AREA1. A constant value C1 fordetermination is preset for the difference comparator 25. AREA1 is arraydata with the same form as L1. Data corresponding to a pixel whoseabsolute value of the difference is larger than C1 is defined as “1”.Data corresponding to a pixel whose absolute value of the difference issmaller than C1 is defined as “0”.

More specifically, for each pixel (x,y),

when |L1[i](x,y)−L1[i−1](x,y)|≧C1,

AREA1(x,y)=“1” and

when |L2[i](x,y)−L1[i−1](x,y)|<C1,

AREA1(x,y)=“0”

The area data AREA1 defined in this way is output by the differencecomparator 25.

The difference comparator 27 receives L2[i−1] and L2[i], calculates theabsolute value of the difference for each pixel, and based on theresult, calculates the area data AREA2. A constant value C2 fordetermination is preset for the difference comparator 27. AREA2 is arraydata with the same form as L2. Data corresponding to a pixel whoseabsolute value of the difference is larger than C2 is defined as “1”.Data corresponding to a pixel whose absolute value of the difference issmaller than C2 is defined as “0”.

More specifically, for each pixel (x,y),

when |L2[i](x,y)−L2[i−1](x,y)|≧C2,

AREA2(x,y)=“1” and

when |L2[i](x,y)−L2[i−1](x,y)|<C2,

AREA2(x,y)=“0”

The area data AREA2 defined in this way is output by the differencecomparator 27.

The difference comparator 30 receives L3[i−1] and L3[i], calculates theabsolute value of the difference for each pixel, and based on theresult, calculates the area data AREA3. A constant value C3 fordetermination is preset for the difference comparator 30. AREA3 is arraydata with the same form as L3. Data corresponding to a pixel whoseabsolute value of the difference is larger than C3 is defined as “1”.Data corresponding to a pixel whose absolute value of the difference issmaller than C3 is defined as “0”.

More specifically, for each pixel (x,y),

when |L3[i](x,y)−L3[i−1](x,y)|≧C3,

AREA3(x,y)=“1” and

when |L3[i](x,y)−L3[i−1](x,y)|<C3,

AREA3(x,y)=“0”

The area data AREA3 defined in this way is output by the differencecomparator 30.

Considering the above explanation, the operation of the adaptivemulti-stage LPF 20 of the second embodiment shown in FIG. 19 will bedescribed below.

The outline will briefly be described first.

The LPF 23 always performs the filtering process in all areas of everyframe. If there is an area where the filtering result by the LPF 23 isinsufficient, the LPF 26 executes the filtering process. If there isstill an area where filtering is insufficient, the filtering process isperformed by the LPF 29 and then by the LPF 31.

It is determined in the following way whether filtering by an LPF issufficient in each area.

The LPFs 22, 24, and 28 filter the preceding frame A[i−1] of the imagedata. The difference between the filtering result of the current frameA[i] of the image data and the filtering result of the preceding frameis compared with a threshold. An area where the difference is largerthan the threshold is determined as an area where filtering by the LPFis insufficient. In contrast, an area where the difference is less thanor equal to the threshold is determined as an area where filtering bythe LPF is sufficient. Information representing the area determined tohave an insufficient difference is sent to the LPF of the next stage.The LPF of the next stage executes the filtering process in that area.The adaptive multi-stage LPF 20 of this embodiment is a 4-stage LPF.Hence, even if the filtering result is finally insufficient, thefiltering result by the LPF 31 is output as the image data L[i] that isthe output of the adaptive multi-stage LPF 20.

The filtering process of this embodiment will be described below inaccordance with the sequence.

First, the input image A[i] and the immediately preceding frame A[i−1]of the image data pass through the LPFs 22 and 23, respectively. Thatis,L1[i]=LPF1(A[i])L1[i−1]=LPF1(A[i−1])Actually, L1[i−1] is already calculated in the output process of theimmediately preceding frame. Hence, the calculated data may be saved andloaded from the memory.

The difference comparator 25 calculates the absolute value of thedifference between L1[i−1] and L1[i], compares the absolute value with athreshold constant value C for each pixel (x,y) to create the area dataAREA1, and sends it to the LPFs 24 and 26.

The LPFs 24 and 26 execute the filtering process LPF2 of the dataL1[i−1] and L1[i] sent from the upstream in the area represented byAREA1. This process can be represented by, for example,L2=L1−(AREA1*L1)+LPF2((AREA1*L1))

(AREA1*L1) means an operation of outputting image data in which pixeldata whose component in AREA1 is “1” is the same as the data in theimage data L1, and pixel data whose component in AREA1 is “0” remains“0”. The thus obtained L2[i] and L2[i−1] are sent to the LPFs 28 and 29.

The LPFs 28 and 29 execute the filtering process LPF3 of the dataL2[i−1] and L2[i] sent from the upstream in the area represented byAREA2. This process can be represented by, for example,L3=L2−(AREA2*L2)+LPF3((AREA2*L2))

(AREA2*L2) means an operation of outputting image data in which pixeldata whose component in AREA2 is “1” is the same as the data in theimage data L2, and pixel data whose component in AREA2 is “0” remains“0”. The thus obtained L3[i] is sent to the LPF 31.

The LPF 31 executes the filtering process LPF4 of the sent data L3[i] inthe area represented by AREA3. This process can be represented by, forexample,L4=L3−(AREA3*L3)+LPF4((AREA3*L3))

(AREA3*L3) means an operation of generating image data in which pixeldata whose component in AREA3 is “1” is the same as the data in theimage data L3, and pixel data whose component in AREA3 is “0” remains“0”.

The thus obtained L[i] is output as image data that is the output of theadaptive multi-stage LPF 20.

The adaptive multi-stage LPF 20 according to the second embodimentincludes the first filter unit (LPFs 23, 26, 29, and 31) and the secondfilter unit (LPFs 22, 24, and 28). Sequentially from the upstream stage,the absolute value of the difference between image data obtained by thenth corresponding filters of the first and second filter units iscalculated. If there is an area where the absolute value of thedifference is less than or equal to a predetermined threshold Cn, onlythat area undergoes the LPF of the next stage. In this way, image datais finally generated, in which the absolute value of the difference issmaller than the predetermined threshold in all areas, and all areashave undergone only the minimum and necessary filtering process. Thisimage data is output as the low-frequency image data L[i] of the frameof interest. In this embodiment, an example in which Cn can be set foreach stage has been described. However, all thresholds Cn may be set toa single common value C. This is a realistic method which can alsoobtain a sufficient effect.

In the example described in the second embodiment, the adaptivemulti-stage LPF 20 as shown in FIG. 17 is used in place of the LPF 1 ofthe first embodiment shown in FIG. 1. However, as shown in FIG. 18, theadaptive multi-stage LPF of the second embodiment and a subtractor maybe combined to form an HPF in place of the HPF of the prior art shown inFIG. 6.

In the second embodiment, a 4-stage LPF is used. The characteristic canbe further improved by using a larger number of stages. In fact, awell-balanced number of stages is preferably set in consideration of,for example, the circuit scale and calculation load. For example, in thesmallest scale, the number of stages=2, as shown in FIG. 20. Even thisarrangement can provide a better function and effect than the prior art.

The adaptive multi-stage low-pass filter preferably executes thefiltering process in accordance with an image portion (e.g., a portionwith fast motion or a portion with slow motion). However, the filteringprocess may be done with an entire image.

FIG. 31 shows another example in which the filtering process isperformed for an entire image.

The DL 21 delays input image data by one frame and outputs it. The DL 21can be formed from a FIFO memory for storing one frame of image data,for example. When the current frame of image data input to the adaptivemulti-stage LPF 20 is A[i], the DL 21 outputs the preceding frame A[i−1]of the image data.

The LPFs 22 and 23 are always active, and filter the input image data inaccordance with the distance constant value d1.

The LPFs 24 and 26 are activated when a determination signal dif_1 fromthe difference comparator 25 is “1” to filter the image data from theupstream LPFs 22 and 23 using the distance constant value d2. If thedetermination signal dif_1 from the difference comparator 25 is “0”, theLPFs 24 and 26 do not perform the filtering process.

The LPFs 28 and 29 are activated when a determination signal dif_2 fromthe difference comparator 27 is “1” to filter the image data from theupstream LPFs 24 and 26 using the distance constant value d3. If thedetermination signal dif_2 from the difference comparator 27 is “0”, theLPFs 28 and 29 do not perform the filtering process.

The LPF 31 is activated when a determination signal dif_3 from thedifference comparator 30 is “1” to filter the image data from theupstream LPFs 28 and 29 using the distance constant value d4. If thedetermination signal dif_3 from the difference comparator 30 is “0”, theLPF 31 does not perform the filtering process.

Let X1 and X2 be pixel data output from the LPFs 22 and 23. Thedifference comparator 25 calculates the absolute value |X1−X2| of thedifference, compares the absolute value with the threshold C1, andoutputs the comparison result as the determination signal dif_1.

More specifically,

when a condition |X1−X2|≦C1 is satisfied, dif_1=“0”, and

when a condition |X1−X2|>C1 is satisfied, dif_1=“1”

This also applies to the difference comparators 27 and 30. Thedifference comparator 27 uses the threshold C2 and outputs thedetermination result as dif_2. The difference comparator 30 uses thethreshold C3 and outputs the determination result as dif_3.

A switch SW1 selects the terminal “1” when the determination signaldif_1 from the difference comparator 25 is “1” and the terminal “0” whenthe determination signal dif_1 is “0”. This also applies to switches SW2to SW4.

Considering the above explanation, the operation of the adaptivemulti-stage LPF 20 shown in FIG. 31, which performs the filteringprocess for an entire image, will be described below.

The outline will briefly be described first.

The LPF 23 always performs the process for every frame. If the filteringresult by the LPF 23 is insufficient, the LPF 26 executes the filteringprocess. If the filtering result is still insufficient, the filteringprocess is performed by the LPF 29 and then by the LPF 31.

It is determined in the following way whether filtering by an LPF issufficient.

The LPFs 22, 24, and 28 filter the preceding frame A[i−1] of the imagedata. The difference between the filtering result of the current frameA[i] of the image data and the filtering result of the preceding frameis compared with a threshold. If the difference is larger than thethreshold, it is determined that the filtering by the LPF isinsufficient. On the other hand, if the difference is less than or equalto the threshold, it is determined the filtering by the LPF of the stageis sufficient, and the filtering process result of that stage is outputas the image data L[i] that is the output of the adaptive multi-stageLPF 20. The adaptive multi-stage LPF 20 is a 4-stage LPF. Hence, if thedetermination result dif_3 of the final difference comparator 30 is “1”,the filtering result by the LPF 31 is output as the image data L[i] thatis the output of the adaptive multi-stage LPF 20.

The filtering process of this embodiment will be described below inaccordance with the sequence.

First, the input image A[i] and the immediately preceding frame A[i−1]of the image data pass through the LPFs 22 and 23, respectively. Thatis,L1[i]=LPF1(A[i])L1[i−1]=LPF1(A[i−1])Actually, L1[i−1] is already calculated in the output process of theimmediately preceding frame. Hence, the calculated data may be saved andloaded from the memory.

The difference comparator 25 calculates the absolute value of thedifference between L1[i] and L1[i−1] and compares the absolute valuewith the threshold constant value C1.

When |L1[i]−L1[i−1]|<C1, the filtering result L1[i] of the current frameis sufficient. Hence, the difference comparator 25 outputs thedetermination signal dif_1=“0”. Consequently, the switch SW1 selects theoutput terminal “0” to output L1[i] as the output L[i] of the adaptivemulti-stage LPF 20. When the determination signal dif_1=“0”, thedifference comparator 27 of the next stage unconditionally sets thedetermination signal dif_2 to “0”. This also applies to the differencecomparator 30.

On the other hand, when |L1[i]−L1[i−1]|≧C1, the filtering result L1[i]of the current frame is insufficient. Hence, the difference comparator25 outputs the determination signal dif_1=“1”. This activates the LPFs24 and 26. The switch SW1 selects the output terminal “1”. The LPFs 24and 26 filter L1[i1] and L1[i−1], respectively.

The difference comparator 27 calculates the absolute value of thedifference between the output L2[i] and L2[i−1] from the LPFs 24 and 26and compares the absolute value with the threshold constant value C2.

When |L2[i]−L2[i−1]|<C2, the filtering result L2[i] of the current frameis sufficient. Hence, the difference comparator 27 outputs thedetermination signal dif_2=“0”. Consequently, the switch SW2 selects theoutput terminal “0” to output L2[i] as the output L[i] of the adaptivemulti-stage LPF 20.

On the other hand, when |L2[i]−L2[i−1]|≧C2, the filtering result L2[i]of the pixel of interest of the current frame is insufficient. Hence,the difference comparator 27 outputs the determination signal dif_2=“1”.This activates the LPFs 28 and 29. The switch SW2 selects the outputterminal “1”. The LPFs 28 and 29 filter L2[i1] and L2[i−1],respectively.

The difference comparator 30 calculates the absolute value of thedifference between the output L3[i] and L3[i−1] from the LPFs 28 and 29and compares the absolute value with the threshold constant value C3.

When |L3[i]−L3[i−1]|<C3, the filtering result L3[i] of the current frameis sufficient. Hence, the difference comparator 30 outputs thedetermination signal dif_3=“0”. Consequently, the switch SW3 selects theoutput terminal “0” to output L3[i] as the output L[i] of the adaptivemulti-stage LPF 20.

On the other hand, when |L3[i]−L3[i−1]|≧C3, the filtering result L3[i]of the current frame is insufficient. Hence, the difference comparator30 outputs the determination signal dif_3=“1”. This activates the LPF31. Each of the switches SW3 and SW4 selects the output terminal “1” tooutput the filtering process result L4[i] by the LPF 31 as the outputL[i] of the adaptive multi-stage LPF 20.

As a result, only one of the outputs from the LPFs 23, 26, 29, and 31 isoutput as the output L[i] of the adaptive multi-stage LPF 20.

The above-described process can be expressed as follows.

The adaptive multi-stage LPF 20 according to the second embodimentincludes the first filter unit (LPFs 23, 26, 29, and 31) and the secondfilter unit (LPFs 22, 24, and 28). Sequentially from the upstream stage,the absolute value of the difference between image data obtained by thenth corresponding filters of the first and second filter units iscalculated. The absolute value of the difference is compared with thepredetermined threshold Cn to determine whether the difference is lessthan or equal to the threshold Cn in all areas. When the determinationis achieved (when it is determined that the difference is smaller thanCn in all areas) for the first time, image data obtained by the nthfilter of the first filter unit is output as the low-frequency imagedata L[i] of the frame of interest.

The operation of the adaptive multi-stage LPF shown in FIG. 31, whichperforms the filtering process for an entire image, has been describedabove. FIG. 32 is a block diagram of an adaptive multi-stage LPF whichperforms the filtering process for an entire image when coefficient=2.

As described above, according to the second embodiment, it is possibleto mitigate the first and second problems.

Third Embodiment

In the third embodiment, a solution to the above-described secondproblem will be described.

In the above-described second embodiment, the distance constant value dof the LPF is appropriately controlled to suppress the ratio of thehigh-frequency component from becoming excessively high. That is, thisprevents the brightness level of display from being saturated (i.e.,SH[i]≧100%, or SH[i]<0 mathematically).

The third embodiment proposes a method of preventing a problem even whensaturation of the display brightness level (i.e., SH[i]≧100%, or SH[i]<0mathematically) still occurs after the above-described process. That is,SH[i] and SL[i] which are calculated once are corrected again to preventany problem.

FIGS. 21 and 22 show a case in which the method of the embodiment isapplied to the prior art in FIG. 6.

In the prior art shown in FIG. 6, SH[i] and SL[i] can be expressed byL[i]=LPF(A[i])H[i]=A[i]−L[i]SH[i]=L[i]+2H[i]SL[i]=L[i]

In the third embodiment, correction to be described later is performedfor SH[i] and SL[i] obtained by calculations up to this point, therebygenerating corrected images SH_0[i] and SL_0[i]. The values of SH_0[i]and SL_0[i] represent brightness levels to be actually displayed. Thatis, both SH_0[i] and SL_0[i] are images that are corrected topractically take a value of 0% to 100%. When a pixel value is expressedby 8 bits, “100%” is the upper limit “255” expressed by 8 bits, and “0%”is literally the lower limit “0” of an 8-bit value.

The parameters in FIGS. 21 and 22 will be defined first. When the valueof a pixel in spatial high-frequency image data mathematically exceeds100%, the exceeding part (excess part) is defined as “SH_Over100”. If SHis mathematically negative, that is, if 2H is a negative value, and theabsolute value of 2H is larger than L, a negative part smaller than thelower limit is defined as “−SH_Under0”.

In the third embodiment, as shown in FIG. 21, when a pixel in spatialhigh-frequency image data has SH_Over100, SH_Over100 is subtracted fromthe pixel and added to a pixel at the same position in spatiallow-frequency image data. That is, SH_Over100 is moved. Similarly, asshown in FIG. 22, to compensate for −SH_Under0, the same value is movedfrom the second sub-frame to the first sub-frame.

This operation can actually be expressed in, for example, the C languageas follows.

  if (SH > 100) {  SH_over100 = SH−100 ; SH_under0 = 0 ; SH = 100 ;}else if (SH < 0) {  SH_over100 = 0 ; SH_under0 = −SH ; SH = 0 ;} else { SH_over100 = 0 ; SH_under0 = 0 ;} SH_0[i] = SH[i] − SH_over100 +SH_under0 ; SL_0[i] = SL[i] + SH_over100 − SH_under0 ;

With this process, if the display brightness level is saturated (i.e.,if SH[i]≧100%, or SH[i]<0 mathematically), the above-describedcorrection can be performed to replace the data with SH_0[i] andSL_0[i]. It is therefore possible to prevent this problem in the image.

FIGS. 23 and 24 show a case in which the method described in the thirdembodiment is applied to the first embodiment in FIG. 1.

In the first embodiment shown in FIG. 1, SH[i] and SL[i] can beexpressed byL[i]=LPF(A[i])H[i]=A[i]−L[i]SH[i]=L[i]+2H[i]SL[i]={L[i]+L[i−1]}/2

In the third embodiment, the correction process is performed for SH[i]and SL[i] obtained by calculations up to this point, thereby obtainingresults SH_0[i] and SL_0[i]. The values of SH_0[i] and SL_0[i] representbrightness levels to be actually displayed and take a value of 0% to100%.

The parameters in FIGS. 23 and 24 will be defined first. When the imagedata SH (=L+2H) mathematically exceeds 100%, the actually exceeding partis defined as “SH_Over100”. If SH is mathematically negative (i.e., if2H is negative, and its absolute value is larger than L), the actualnegative part is defined as “−SH_Under0”.

In the third embodiment, as shown in FIG. 23, half of SH_Over100 ismoved from the first sub-frame to the second sub-frame. Additionally,the remaining half is moved to the immediately preceding secondsub-frame.

Similarly, as shown in FIG. 24, to compensate for −SH_Under0, a valuecorresponding to half of the value is moved from the second sub-frame tothe first sub-frame of interest. Additionally, a value corresponding tothe remaining half is moved from the immediately preceding secondsub-frame to the first sub-frame of interest.

This operation can actually be expressed in, for example, the Clanguage, as follows.

  if (SH > 100) {  SH_over100 = SH-100 ; SH_under0 = 0 ; SH = 100 ;}else if (SH < 0) {  SH_over100 = 0 ; SH_under0 = −SH ; SH = 0 ;} else { SH_over100 = 0 ; SH_under0 = 0 ;} SH_0[i] = SH[i] − SH_over100 +SH_under0 ; SL_0[i] = SL[i] + (1/2) * SH_over100 − (1/2) * SH_under0 ;SL_0[i−1] = SL[i] + (1/2) * SH_over100 − (1/2) * SH_under0 ;

With this process, if the display brightness level is saturated (i.e.,if SH[i]≧100%, or SH[i]<0 mathematically), the above-describedcorrection can be done to replace the data with SH_0[i] and SL_0[i]. Itis consequently possible to prevent this problem in the image. Even whenthe method described in this embodiment is applied, the temporal centerof gravity of the spatial high-frequency component and that of thespatial low-frequency component do not change. Hence, no shift occursbetween the display position of the spatial high-frequency component andthat of the spatial low-frequency component in an image obtained inpursuit. It is therefore possible to prevent any operation error causedby the saturation of the brightness level while preventing distortionsuch as tail-blurring in an image observed in pursuit.

Fourth Embodiment

In the fourth embodiment, an example of a solution to theabove-described first and second problems will be described.

In the fourth embodiment, an input image is separated into a pluralityof spatial frequency components, and the display level is distributed(multiplied) to a plurality of sub-frames so that the temporal centersof gravity of the components match. This improves motion blur, doubleblurring and flicker. In the above-described first to third embodiments,one input frame is subjected to double-rate conversion (i.e., separatedinto two sub-frames), and spatial frequency separation is done toseparate image data into a spatial high-frequency component and aspatial low-frequency component.

Let N be the magnification of the number of frames, and R be the numberof frequency bands for spatial frequency separation. In theabove-described embodiments, the frame frequency is doubled (N=2), orthe spatial frequency band is separated into two bands (R=2), that is, ahigh-frequency component and a low-frequency component.

However, the present invention does not limit the number of sub-frames(i.e., does not limit N of “Nx-rate” to “2”). In spatial frequency bandseparation as well, the present invention does not limit the number ofbands (i.e., does not limit R of “R bands” to “2”).

In this embodiment, conditions for obtaining the same effect asdescribed above even when the frame frequency of an input image ismultiplied by N (or separated into N sub-frames), and the input image isseparated into R spatial frequency bands will be described. N and R arepositive integers greater than or equal to 2.

N is not limited to 2 and can be 3 or more to obtain the same effect. Ifanything, large N is preferable for satisfying the condition of thepresent invention because the degree of freedom in distribution is high.

R is designed such that when an input image is separated into R spatialfrequency bands, two adjacent bands (e.g., J and (J+1)) always have theabove-described relationship for R=2. In this case, the samerelationship holds in all bands from the first to Rth bands. That is,the R band components are made to have such a relationship that thetemporal distribution of the band component images becomes smaller indescending order of spatial frequency (in other words, the temporaldistribution becomes larger in ascending order of spatial frequency),and the temporal centers of gravity of all band components match. Atthis time, the effect of the present invention, that is, the effect ofimproving motion blur, preventing flicker, and reducing distortion suchas tail-blurring is obtained. That is, in the present invention, evenwhen N and R are generalized, the same effect as in the above-describedarrangements with N=2 and R=2 is obtained.

Even in spatial frequency separation, the number of bands is notlimited. In the fourth embodiment, N=5. This is equivalent to a processof, for example, converting the frame rate of an image having a framefrequency of 24 Hz (generally corresponding to a moving image of amovie) into a 5× frame rate of 120 Hz. The number R of bands will beexemplified as R=3 which does not make the scale of circuit andcalculation so large. That is, an example in which the spatial frequencyis separated into three bands: High, Middle, and Low will be described.

FIG. 26 is a block diagram of the fourth embodiment. FIG. 25 shows thecorrection intensities (multiplier coefficients) of the display level ofthe spatial frequency band of each sub-frame according to the fourthembodiment.

The operation of the arrangement in FIG. 26 will be described below. Afilter 40 receives image data A[i] of one frame at a frame rate of 24Hz. The filter 40 separates the input image data into three spatialfrequency band components H, M, and L based on two distance constantvalues d1 and D2 (e.g., d1=3 pixels, and d2=8 pixels). The componentsare temporarily saved in frame memories 42 to 44.

A frame sync signal quintupling circuit 41 quintuples the frame syncsignal of the input image A[i] to generate and output clk5.

Each of switches 45 and 46 is reset by the frame sync signal of theimage A[i] and selects the output terminal in the order of(1)→(2)→(3)→(4)→(5) in accordance with clk5.

The H component temporarily saved in the frame memory 42 is read outfive times in accordance with clk5 and sent to the switch 45. The switch45 is connected to the output terminal (1) at the first stage of clk5.The output terminal then switches to (2), (3), (4), and (5) in thisorder. The M component saved in the frame memory 43 is also read outfive times. The output terminal of the switch 46 sequentially switchesfrom (1) to (5). The L component temporarily saved in the frame memory44 is also read out five times.

Multipliers GH1 to GH5 which multiply the high-frequency components bytheir gains {0, 0.5, 4.0, 0.5, 0} shown in FIG. 25 are providedimmediately after the output terminals of the switch 45.

Similarly, multipliers GM1 to GM5 which multiply the M components bytheir gains {0, 1.5, 2.0, 1.5, 0} shown in FIG. 25 are providedimmediately after the output terminals of the switch 46.

That is, in reading out a component N times (in the embodiment, N=5),the gain (multiplier coefficient) gradually becomes large. The gain ismaximized at the intermediate time (in the embodiment, third time) andthen gradually becomes small.

A multiplier using a gain {1, 1, 1, 1, 1} is provided immediately afterthe frame memory 44. The gain of the low-frequency component does notchange. Hence, one multiplier suffices, as shown in FIG. 26.

Actually, the same arrangements as the switches 45 and 46 and thesucceeding multipliers are necessary even immediately after the framememory 44. However, the arrangements are not shown because thecoefficient does not change in this case.

Adders 47 to 49 add the multiplied image data. As a result, S1[i],S2[i], S3[i], S4[i], and S5[i] are output from the first to fifth stagesof clk in this order and displayed, which are given byS1[i]=L[i]S2[i]=L[i]+1.5M[i]+0.5H[i]S3[i]=L[i]+2M[i]+4H[i]S4[i]=L[i]+1.5M[i]+0.5H[i]S5[i]=L[i](where [i] is the frame number of the corresponding input image A[i])

In this embodiment, the plurality of spatial frequency band componentsare set such that the temporal distribution becomes smaller indescending order of spatial frequency (in other words, the temporaldistribution becomes larger in ascending order of spatial frequency),and the temporal centers of gravity of all components match, asdescribed above. FIG. 25 complies with the conditions. The coefficientsshown in FIG. 25 are bilaterally symmetrical for all the H, M, and Lcomponents. The temporal center of gravity is located at the center(third sub-frame), and the temporal centers of gravity of all componentsmatch. Weight concentration on a portion close to the intermediate frameis most conspicuous among the coefficients of the H component. Therelationship of the degree of concentration is represented by H>M>L. Asfor the L component, the weight is flat. That is, the temporaldistribution becomes smaller in descending order of spatial frequency(larger in ascending order of spatial frequency), as can be seen. Thetemporal distribution can be expressed in detail in the following wayusing the example of the coefficient array in FIG. 25. The ratio of theH coefficient to the M coefficient increases toward the third sub-frameat the center and is maximized at the third sub-frame. The ratio of theM coefficient to the L coefficient also increases toward the thirdsub-frame at the center and is maximized at the third sub-frame. Therelationship between the L and M coefficients is also the same. FIG. 25shows an example of a coefficient array that can be generalized in thisway.

In this display, when an object that moves in an image is pursued, theH, M, and L components of the observed image have no spatial shift.Hence, an image free from distortion such as ghosting or tail-blurringis observed.

The four embodiments of the present invention have been described above.In the embodiments, the output destination of each of the finallyobtained Nx-rate frames is not mentioned specifically. The outputdestination can be either a hold- or impulse-type display device. Theoutput destination is not limited to a display device. Instead, the datamay be stored in a storage medium such as a DVD or a storage device suchas a hard disk as a moving image file.

Like the above-described modification of the first embodiment, thesecond and subsequent embodiments also allow a computer program toimplement an equivalent process.

Normally, a computer program is stored in a computer-readable storagemedium such as a CD-ROM. The storage medium is set in the read device(e.g., CD-ROM drive) of a computer, and the program is made executableby copying or installing it in a system. Hence, such a computer-readablestorage medium is also incorporated in the present invention.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2007-207181, filed Aug. 8, 2007, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image processing apparatus comprising: one ormore processors; and a memory for storing programs to be executed by theone or more processors, wherein when the programs stored in the memoryare executed by the one or more processors, the one or more processorsfunctions as: an inputter configured to input an input image framecorresponding to moving image data; a first obtainer configured toobtain spatial low-frequency component image data from the input imageframe input by the inputter; a second obtainer configured to obtainspatial high-frequency component image data from the input image frameinput by the inputter; a generator configured to generate a first outputimage frame based on both of second spatial low-frequency componentimage data obtained by the first obtainer from a second input imageframe input by the inputter and first spatial low-frequency componentimage data obtained by the first obtainer from a first input image frameinput by the inputter, and generate a second output image frame based onthe spatial high-frequency component image data obtained by the secondobtainer from the first input image frame input by the inputter; and anoutputter configured to output the first and second output image framesgenerated by the generator for displaying.
 2. The image processingapparatus according to claim 1, wherein the generator generates thesecond output image frame by using both of first low-frequency componentimage data obtained by applying filter processing to the first inputimage frame and the first input image frame, and generates the firstoutput image frame by using both of second low-frequency component imagedata obtained by applying the filter processing to the second inputimage frame and the first low-frequency component image data, whereinthe first input image frame being subsequent to the second input imageframe and the first and second input image frames are input by theinputter.
 3. The image processing apparatus according to claim 1,wherein the first obtainer is configured to obtain the spatiallow-frequency component image data by applying low-pass filterprocessing for cutting of a spatial frequency component higher than athreshold to the input image frame.
 4. The image processing apparatusaccording to claim 1, wherein the second obtainer is configured toobtain the spatial high-frequency component image data by applyinghigh-pass filter processing for cutting of a spatial frequency componentlower than a threshold to the input image frame.
 5. The image processingapparatus according to claim 1, wherein the outputter outputs the firstoutput image frame after outputting the second output image frame fordisplaying.
 6. The image processing apparatus according to claim 1,further comprising a correction unit configured to, in a case where apixel value of the second output image frame exceeds a predeterminedthreshold pixel value, add the exceeded value to a pixel value of thefirst output image frame, wherein the outputter outputs the correctedfirst output image frame and the second output image frame.
 7. The imageprocessing apparatus according to claim 1, wherein the generatorgenerates the second output image frame based on both of the spatiallow-frequency component image data obtained by applying filterprocessing to an input image frame and image data according todifference between the spatial low-frequency component image data andthe input image frame, and generates, an output image frame to be outputbefore outputting the second output image frame and output image frameto be output after outputting the second output image frame, based onthe spatial low-frequency component image data.
 8. An image processingmethod comprising: inputting an image frame corresponding to movingimage data; obtaining, in a first obtain function, spatial low-frequencycomponent image data from the input image frame input in the inputting;obtaining, in a second obtain function, spatial high-frequency componentimage data from the input image frame input in the inputting; generatinga first output image frame based on both of second spatial low-frequencycomponent image data obtained by the first obtain function from a secondinput image frame input in the inputting and first spatial low-frequencycomponent image data obtained by the first obtain function from a firstinput image frame input in the inputting and generating a second outputimage frame based on the spatial high-frequency component image dataobtained in the second obtain function from the first input image frameinput in the inputting; and outputting the generated first and secondoutput image frames for displaying wherein the method is implemented byone or more processors executing programs stored in a memory, theprograms including code for performing the image processing method. 9.The image processing method according to claim 8, wherein the secondoutput image frame is generated by using both of first low-frequencycomponent image data obtained by applying filter processing to the firstinput image frame and the first input image frame, and the first outputimage frame is generated by using both of second low-frequency componentimage data obtained by applying the filter processing to the secondinput image frame and the first low-frequency component image data,wherein the second input image frame being subsequent to the first inputimage frame and the first and second input image frames are input in theinputting.
 10. The image processing method according to claim 8, furthercomprising, in a case where a pixel value of the second output imageframe exceeds a predetermined threshold pixel value, adding, theexceeded value to a pixel value of the first output image frame, whereinthe processed first output image frame and the second output image frameare output for displaying.
 11. The image processing method according toclaim 8, wherein the second output image frame is generated based onboth of the spatial low-frequency component image data obtained byapplying filter processing to an input image frame and image dataaccording to a difference between the spatial low-frequency componentimage data and the input image frame, and an output image frame to beoutput before outputting the second output image frame and output imageframe to be output after outputting the second output image frame aregenerated based on the spatial low-frequency component image data.
 12. Anon-transitory computer-readable medium storing a program which, whenexecuted by one or more processors, causes the one or more processors toperform to execute an image processing method, the image processingmethod comprising: inputting an image frame corresponding to movingimage data; obtaining, in a first obtain function, spatial low-frequencycomponent image data from the input image frame input by the inputting;obtaining, in a second obtain function, spatial high-frequency componentimage data from the input image frame input by the inputting; generatinga first output image frame based on both of second spatial low-frequencyimage data obtained in the first obtain function from a second inputimage frame input in the inputting and first spatial low-frequencycomponent image data obtained in the first obtain function from a firstinput image frame input by the inputting, and generating a second outputimage frame based on the spatial high-frequency component image dataobtained in the second obtain function from the first input image frameinput in the inputting; and outputting the generated first and secondoutput image frames for displaying.
 13. The medium according to claim12, wherein the second output image frame is generated by using both offirst low-frequency component image data obtained by applying filterprocessing to the first input image frame and the first input imageframe, and the first output image frame is generated by using both ofsecond low-frequency component image data obtained by applying thefilter processing to the second input image frame and the firstlow-frequency component image data, wherein the second input image framebeing sequential to the first input image frame and the first and secondinput image frames are input in the inputting.
 14. The medium accordingto claim 12, further comprising adding, in a case where a pixel value ofthe second output image frame exceeds a predetermined threshold pixelvalue, the exceeded value to a pixel value of the first output imageframe, wherein the processed first output image frame and the secondoutput image frame are output for displaying.
 15. The medium accordingto claim 12, wherein the second output image frame is generated based onboth of the spatial low-frequency component image data obtained byapplying filter processing to an input image frame and image dataaccording to difference between the spatial low-frequency componentimage data and the input image frame, and an output image frame to beoutput before outputting the second output image frame and output imageframe to be output after outputting the second output image frame aregenerated based on the low-frequency component image data.
 16. The imageprocessing apparatus according to claim 1, wherein a first generator isconfigured to generate the first output image frame by using the spatiallow-frequency component image data obtained from the plurality of inputimage frame, and wherein a second generator is configured to generatethe second output image frame by using the spatial high-frequencycomponent image data obtained from one input image frame.
 17. The imageprocessing apparatus according to claim 1, wherein the generator doesnot use second spatial high-frequency component image data obtained bythe second obtainer from the second input image frame to generate thesecond output image frame.